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DETAILED ACTION 
Claim Objections 

1. Claim 26 is objected to because of the following informalities: video sequences do not 
comprise MPEG-3 format because MPEG-3 is used for audio compression, MP3. Appropriate 
correction is required. 

Claim Rejections - 35 USC§ 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 1-32 are rejected under 35 U.S.C. 103(a) as being unpatentable over Kuchinsky et 
al ("FotoFile: A Consumer Multimedia Organization and Retrieval System", CHI 1999, 
Pittsburgh PA USA, ACM 1999 0-201-48559-1, pages 496-503) in view of Lang et al 
("'Looking at Pictures: Affective, facial, visceral, and behavioral reactions" Psychophysiology, 
1993, pages 261-273) and further in view of Essa ("Facial Expression Recognition using a 
Dynamic Model and Energy", International Conference on Computer Vision '95, Cambridge, 
MA, June 20-23, 1995) 

a. Referring to Claim 1, Kuchinsky teaches retrieving from storage a set of photos to 
be transmitted to said viewer based on an identifier associated with the mood 
(Kuchinsky at page 498, right column, lines 7-12, teaches using annotating 
content as retrieval attributes, and such annotating contents is associated with the 
viewer's emotional characteristics, page 499, left column, line 10. The annotating 
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contents are also called metadata attributes, which are used to tag photo pictures, 
page 498, left column, lines 12-14), and transmitting the set of photos retrieved to 
said viewers for display in the form of an electronic photo album (page 498, left 
column, lines 23-25). However, Kuchinsky does not teaches the tag identifier is 
an emotional identifier, where the emotional identifier is related to the viewer's 
mood, and the mood of determined by the facial expression of the viewer on an 
image taken by a camera, and the image is subjected to a pattern recognition 
module of a processor. Lang teaches the viewer's emotional interests/mood at 
pictures can be measured based on the viewer's facial action, page 266, right 
column, lines 1-12. However, Lang does not teach the facial action/expression is 
determined by analysis of a facial expression image of said viewer taken by a 
camera, by a pattern recognition module of a processor. Essa teaches using a 
pattern recognition module of a processor to analyze facial expression images, 
and determine a person's mood (see figure 8 in "Facial Expression Recognition 
using a Dynamic Model and Energy", and section 5.2). At the time the invention 
was made, a person of ordinary skill in the art would have been motivated to use a 
mood/emotion of a person when viewing a picture as the metadata attribute for 
later retrieval purpose because Kuchinsky teaches the annotating contents can be 
used as retrieval attributes, and the annotating content can be anything as long as 
it matches the user's expectations (page 498, left column, line 26) and keeps the 
annotating process enjoyable (page 498, right column, line 25), which means the 
annotating content can be user's/viewer's mood/emotional interest at a picture 
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through the facial expression, and it certainly keeps the annotating process 
automated and enjoyable as requested by Kuchinsky. In addition, a person of 
ordinary skill in the art would also be motivated to use Essa's pattern recognition 
module to analyze a viewer's mood/emotional interest when he/she is viewing a 
picture because Lang asks the facial actions to be accurately measured to rate the 
interest of a person at a picture, and Essa's pattern recognition module precisely 
provides the need by scaling the facial expression, such as smile, sadness or etc, 
in numbers as shown in figure tables on page 7, so a system can correctly rate the 
interest of the person at the picture. 

b. Referring to Claim 2, the combination of Kuchinsky, Lang and Essa teaches 
wherein the set of photos is transmitted over the Internet to said viewer, and said 
electronic photo album comprising a virtual photo album (see Kuchinsky, page 
499, figure 1, and left column, line 1 1). 

c. Referring to Claim 3, the combination of Kuchinsky, Lang and Essa teaches 
wherein the pattern recognition module is locally attached computer (the pattern 
recognition module in Essa is an independent system module, which can be 
attached to a local computer or a remote server). 

d. With regard to Claim 4, see explanation in Claim 3. 

e. Referring to Claim 5, the combination of Kuchinsky, Lang and Essa teaches 
wherein the previously stored images are facial expressions of the viewer (Essa 
teaches there are 5 facial expression of a person used as templates for comparison, 
Essa, page 7, lines 11-12). 
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f. Referring to Claim 6, the combination of Kuchinsky, Lang and Essa teaches 
providing the previously stored images of the facial expression of the viewer with 
emotional identifiers (Kuchinsky teaches the annotating contents are used as 
retrieval attributes, and since the facial expression are used as annotating contents, 
of course, there are previously stored image of facial expression of the viewer 
with emotion identifiers stored in a database memory waiting for retrieval). 

g. Referring to Claim 7, the combination of Kuchinsky, Lang and Essa teaches 
wherein the emotional identifier are supplied by the viewer to designate a 
particular mood for the facial expression for each respective image of said 
plurality of previously stored image (Essa teaches the facial expression of the 
each image represents a particular mood for the viewer, see figure 9 in Essa). 

h. With regard to Claim 8, see explanation in Claim 5. 

i. With regard to Claim 9, see explanation in Claim 6. 
j. With regard to Claim 10, see explanation in Claim 7. 

k. Referring to Claim 11, the combination of Kuchinsky, Lang and Essa teaches 
capturing a second image of a facial expression of the viewer^ subsequent to said 
viewer looking at least at one photo of said set of photos transmitted; updating the 
set of photos transmitted according to the second image of the facial expression 
captured (Essa teaches there are 5 different facial expression templates, and Lang 
teaches the rated interest of a person at different pictures is different because of 
different facial expressions as result of different reactions at the pictures, 
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therefore, the attribute tags in Kuchinsky will be different for different photo 
pictures). 

1. Referring to Claim 12, the combination of Kuchinsky, Lang and Essa teaches 
wherein the image of the facial expression is captured as one of a JPEG image, 
TIFF image, BMP image and PCX image (the above mentioned image formats 
are the most popular standard image formats available for image processing). 

m. Referring to Claim 13, the combination of Kuchinsky, Lang and Essa teaches 
wherein the image of the facial expression is a video sequence (Essa teaches the 
image of the facial expression is a video sequence). 

n. Referring to Claim 14, the combination of Kuchinsky, Lang and Essa teaches 
wherein the said video sequence comprises an MPEG sequence (the standard 
video image format is MPEG). 

o. Referring to Claim 15, the combination of Kuchinsky, Lang and Essa teaches 
wherein modifying the emotional identifier associated with the mood of the 
viewer according to at least one of local weather; other people adjacent to the 
viewer; illumination adjacent to the viewer; time of day; date; and preferred 
activities of the viewer (Kuchinsky at page 500, left column, lines 1-4, teaches 
modifying the identifier based on faces of new people adjacent to the viewer). 

p. Referring to Claim 16, the combination of Kuchinsky, Lang and Essa teaches 
wherein includes transmitting an order of the set of photos according to the 
modification to the emotional identifier (Kuchinsky teaches using the confirmed 
attribute tag to retrieve photo pictures). 
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q. With regard to Claim 17, see explanation in Claim 1, the only additional 
limitation is a display for display set of photos to the viewer as a virtual photo 
album (see figure 1 in Kuchinsky). 

r. Referring to Claim 18, the combination of Kuchinsky, Lang and Essa teaches 
wherein said user interface means includes feedback means for transmitting 
subsequent images of said viewer's facial expression after a display of at least one 
photo of said set photos (Kuchinsky' s user interface means includes feedback 
means for transmitting annotating contents after photo pictures are displayed); 
wherein said pattern recognition module of said processor compares said 
subsequent images of said viewers facial expression for determining the mood of 
the viewer according tot said subsequent images (Essa teaches the pattern 
recognition module analyzes the facial expressions to determine the mood of a 
viewer at pictures in Lang, and the mood/emotional interest is used as annotating 
content/retrieval attributes in Kuchinsky' s retrieval system to retrieve photo 
pictures in a photo album); and said retrieval means retrieves a respective set of 
electronic photos corresponding to the mood of the viewer determined according 
to each of said subsequent images of the viewer's facial expression. 

s. With regard to Claim 19, see explanation in Claim 12. 

t With regard to Claim 20, see explanation in Claim 12. 

u. With regard to Claim 2 1 , see explanation in Claim 2. 

v. With regard to Claim 22, see explanation in Claim 2. 

w. With regard to Claim 23, see explanation in Claim 1 for virtual photo album. 
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x. With regard to Claim 24, see explanation in Claim 1 for virtual photo album, 
y. With regard to Claim 25, see explanation in Claim 13. 

z. With regard to Claim 26, MPEG, MPEG-1, MPEG-2, and MPEG 4 are the 
standard compression formats for video sequence (note there is no MPEG-3, 
which is MP3 used for audio compression). 

aa. Referring to Claim 27, the combination of Kuchinsky, Lang and Essa teaches 
wherein said user interface means include means for initially inputting photos of 
facial expressions of said viewer for stages in said storage area of said processor, 
each one of said photos having an associated emotional identifier that indicates a 
mood represented by each representative facial expression of said each one of said 
photos (Kuchinsky teaches a storage means for storing the photo pictures and the 
metadata attributes used to tag the photo pictures, and since the attributes in 
Kuchinsky now refers to the facial expression of the viewer as taught by the 
combination of Lang and Essa, therefore, the facial expressions as attributes are 
stored in the storage means as well). 

bb. With regard to Claim 28, see explanation in Claim 27. 

cc. With regard to Claim 29, see explanation in Claim 27. 

dd. With regard to Claim 30, see explanation in Claim 27. 

ee. With regard to Claim 31, see explanation in Claim 27. 

ff. With regard to Claim 32, see explanation in Claim 27. 
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Conclusion 

3. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

a. Matraszrek et al, US Publication No. 2003/01 17651 Al, see figures 2-4C. 

b. Fedorovskaya et al, U.S. Publication No. 2003/0156304 Al, see figures 2-6B. 

c. Parulski et al, U.S. Patent No. 6,629,104 Bl, see figure 1. 

d. Black et al, "Tracking and Recognizing Rigid and Non-rigid Facial Motions using 
Local Parametric Models of Images Motions", IEEE, ISBN 0-8186-7042-8, pages 
374-381, 1995. 

4. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Tom Y Lu whose telephone number is (703) 306-4057. The 
examiner can normally be reached on 8:30AM-5PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Leo H Boudreau can be reached on (703) 305-4706. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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